Relevance & Research Question: Satisficing behavior is a widespread hazard in Web surveys because interview supervision is limited in absence of a human interviewer. Therefore, it is important to devise methods which help to identify and to mitigate satisficing. The paper examines whether innovative questionnaire design can be an efficient means to reduce non-substantial answers, non-differentiation in matrix questions, and speeding. We analyze to what extent these types of satisficing can be reduced through three tools suggested in recent research. First, several studies used prompts to reduce the incidence of non-substantial answers. Second, some authors proposed alternative designs for matrix questions (so-called scrolling matrix questions) to mitigate response non-differentiation. Third, control questions (or instructional manipulation checks) are intended to identify inattentive respondents. Our contribution provides further evidence on how these tools are suited to reduce satisficing response behavior and to increase the quality of the respondents’ answers.
Methods & Data: For our analyses, we use data from two Web surveys with 1,000 and 2,000 respondents, respectively. In the first sample, drawn from a probability-based online panel, half of the respondents were prompted when providing non-substantial answers. In the second survey, drawn from a non-probability online panel, each of the design innovations is randomly assigned to half of the sample. The experimental groups are compared with the control groups using t-tests or chi²-tests. A multivariate regression model for satisficing behavior is estimated to test whether the design innovations contribute to the explanation of satisficing if we control for respondent characteristics which are predictive of satisficing.
Results: Preliminary analyses show that prompts are a well-suited tool to reduce item nonresponse in Web surveys. However, this might come at the expense of increased survey breakoff. Since the second survey fields in end of December 2012, further results will be available by end of January 2013.
Added Value: Satisficing response behavior is a major concern in Web surveys. We assess the potentials of three easily implementable tools to increase data quality and discuss their advantages and pitfalls. To the extent that satisficing varies between survey modes, mixed-mode survey can particularly benefit from these measures.